1. Field of the Invention
The present invention relates to technologies for adjusting the white balance of a captured image. In particular, it relates to technologies for adjusting the white balance of an image that has been captured by an image capture device that is provided with an illumination portion (flash emission device) that allows the flashed light to be controlled.
2. Description of the Related Art
Image capture devices such as digital still cameras use a flash emission device in order to illuminate an object to be captured with a flashed light as a helper light, when the light that is reflected from the object to be captured during imaging is weak. Image capture devices may also employ a flash emission device as backlight correction in order to illuminate a flashed light on a person (captured object) to brighten the person, and in doing so keep the person from appearing dark when that person is imaged against a bright background such as sunlight.
On the other hand, there are instances when capturing an image of an object to be captured after a flash emission device has emitted a flashed light results in an photograph (captured image) with an unnatural color balance with respect to the captured object image with light (illumination light) due to the lighting originally present in the image capture environment (the captured object image on the captured image that is obtained from the illumination light that is reflected off of the captured object) and the captured object image due to the flashed light (that captured object image on the captured image that is obtained from the flashed light that is reflected by the object to be captured). This is because there is a difference in the color temperature between the illumination light that originally illuminates the captured object and the flashed light.
Image capture devices such as digital still cameras generally perform processing known as white balance (WB) adjustment (hereinafter, WB adjustment). An example of this would be a case in which the object to be captured is illuminated by lighting with a low color temperature, such as an incandescent lamp, in which case adjustment is performed to relatively weaken the red (R) component signal (image data values) of the captured image data, and conversely, to strengthen the blue (B) component signal (image data values), in order to express white in the object to be captured as white on the captured image. WB adjustment can remove the effects of the illumination light, so that white objects are white objects in the captured image. Methods for finding the color temperature of the lighting include using a color sensor, estimating from the color distribution of the image data that are captured (captured image data), and selection by the user. When the image capture device captures an image using a flashed light, it can perform WB adjustment in accordance with the color temperature of the flashed light, which is already known.
However, when an image is captured using a flashed light, in many situations light other than the flashed light, that is, the illumination light (external light) originally present in the image capture environment, blends with the flashed light, and together the two illuminate the object to be captured. For this reason, performing WB adjustment based on the flashed light results in natural correction for objects that are brightly illuminated by the flashed light, however, appropriate WB adjustment is not performed for objects that are poorly lit by the flashed light or objects that are not illuminated by the flashed light at all (such as background), and this leads to an unnatural captured image (areas originally white become colors other than white in the captured image).
The image capture device disclosed by Japanese Patent Publication 3540485 is an example of a conventional image capture device for solving this problem. Japanese Patent Publication 3540485 discloses a configuration for an image capture device (electric still camera) that finds, per pixel, the ratio of pixel values of two images that are captured with and without a flashed light and determines the contribution of the flashed light based on the value of this ratio, and then, based on the results of this determination, selects a WB coefficient for the external light, a WB coefficient for the flashed light, and WB coefficients between these two, and performs WB adjustment. When “a” is the pixel value of the image captured with the emission of a flash and “b” is the pixel value of the image captured without the emission of a flash, the ratio (flashed light contribution)=a/b. With the device of Japanese Patent Publication 3540485, when a/b≈1 it is determined that that pixel is in a region not reached by the flashed light, and when a/b>>1 it is determined that that pixel is in a region that is sufficiently reached by the flashed light. Japanese Patent Publication 3540485 further discloses that by restricting the variability of the WB coefficient for each pixel, it is possible to keep the WB coefficient from changing suddenly between pixels even when the captured object has moved between the capturing of the two images.
However, the structure of the conventional image capture device has the following three problems.
The first is that with the conventional image capture device, a color pseudo-border occurs in the region where the WB coefficients are switched.
In Japanese Patent Publication 3540485, a WB coefficient for the external light, a WB coefficient for the flashed light, and a WB coefficient between the two are “selected” based on the results of “determination” of the flashed light contribution, but this intermediate WB coefficient must be a “discrete value.” Thus, a color pseudo-border occurs in the region in which the WB coefficient is switched when the ratio of both the external light and the flashed light are continuously altered while irradiating the object to be captured. This is because the flashed light contribution is “non-linear.” This is explained below.
First, the flashed light contribution is shown to be “non-linear.”
For each pixel, when C is the external light illumination, B is the flashed light illumination, R is the reflectance, and “a” and “b” are the pixel values of pixels with and without the emission of the flashed light, the pixel values a and b become:a=(C+B)·R,b=C·R and the flashed light contribution K isK=a/b=1+B/C. Thus, the flashed light contribution K is a “non-linear” parameter (value range=1 to ∞) in which the value increases sharply the smaller the external light illumination C.
Thus, to perform linear WB adjustment with respect to a change in the external light illumination C, it is necessary to partition more narrowly the smaller the range of values of the external light illumination C, and a large number of values must be readied in a table or the like.
Further, not only it is necessary to partition based on the range of values for the external light illumination C, but there are various types of external lighting (clear skies, fluorescent light, light bulbs, etc.), and a WB coefficient must be readied for each combination thereof, and thus significant circuitry and memory are needed.
In instances where the object to be captured has moved, attempting to continuously change the WB coefficient in the regions where it has moved (regions on the captured image) and the regions where it has not moved (regions on the captured image) similarly requires many WB coefficients.
As illustrated above, it is not realistic to solve the problem by increasing the number of WB coefficients, and thus intermediate WB coefficients must be made “discrete values,” resulting in color pseudo-borders in regions where the WB coefficient is switched.
Secondly, with the conventional image capture device, the precision of WB adjustment is poor, and information dropping precludes complete WB adjustment.
The region of the flashed light contribution K (=a/b) is 1 to ∞, and since it is not possible to handle the value ∞, the flashed light contribution K must be restricted to a predetermined upper limit value th.
Captured objects where K≧th are all corrected with the same WB coefficient when the upper limit value th is set to a low value, and thus the precision of WB adjustment drops. A flash is often emitted primarily in order to brighten objects that are dark, and thus the external light illumination C tends to be small and the flashed light illumination B tends to be large. The flashed light contribution K (=1+B/C) thus becomes a large value, making K≧th and lowering the precision of WB adjustment.
On the other hand, when the upper limit value th is set to a high value, a high degree of bit precision is necessary in order to maintain accuracy within a small-value region because the flashed light contribution is a “non-linear” parameter that increases abruptly, and this increases the scale of the circuitry.
Further, truncation leads to information loss and complete WB adjustment thus is not possible in the first place.
Thirdly, with the conventional image capture device, shifts in color stand out in regions where the captured object has moved.
Movement of the captured object while capturing two images with and without a flash shifts the flashed light contribution that is calculated from the two images, and this results in inappropriate WB adjustment in regions where there is movement and is expressed as a shift in color.
In the present application, this problem is solved by using a low pass filter (LPF) as described later, but in general the linear nature of a filter does not allow an LPF to be adopted for the “non-linear” flashed light contribution of Japanese Patent Publication 3540485, and thus cannot reduce color shifting.
Specifically, the flashed light contribution increases sharply the smaller the external light illumination C. The value after LPF thus is strongly affected by neighboring pixels that have extremely large values, and this precludes appropriate LPF processing.
It is conceivable to use a non-linear LPF in order to change the weighting coefficient, for example, in accordance with the pixel value. Changing the weighting coefficient with high precision, however, similarly requires weighting coefficients to be readied in a table, etc., and increases the scale of the circuit too much, and thus is not practical.